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ABSTRACT 



The invention describes a scheme by which the dynamic 
characteristics of virtual links provided to a first network 
protocol engine are fed back to a second network protocol 
engine, and used to improve the performance of the second 
protocol traffic across a backbone. As an example, the 
invention describes a scheme by which the dynamic char- 
acteristics of the virtual links provided to the HPR network 
are fed back to the RTP protocol, and used to improve the 
performance of RTP traffic across the IP backbone. The 
invention provides this feedback using the mechanisms 
provided by the HPR networking architecture, and thus can 
inter-operate with HPR networks that do not support this 
invention. In one embodiment of the present invention, on 
the link characteristics are determined by programs running 
on both end of the IP network. These link characteristics are 
sent to the HPR end-node when an HPR connection is 
established. The HPR end-node obtains the information 
during the connection establishment phase. During the ini- 
tiation of a connection, the HPR connection establishment 
protocols send out route-setup messages along the desired 
path of the connection. When the message is received by the 
node along the path, it is required to send its fixed bandwidth 
and delay in the route-setup control vector, a component of 
the set-up message. This value is dynamically changed so as 
to reflect the current state of the network. Three methods to 
obtain the current link characteristics of the multi-hop IP 
network are described. 

12 Claims, 9 Drawing Sheets 
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DYNAMIC PARAMETER ESTIMATION FOR protocol terminates the SNA connection at the boundary of 

EFFICIENT TRANSPORT OF HPR DATA ON me SNA and me IP networks. And splices the two resulting 

jp SNA connections by means of a TCP connection. The 

splicing imposes a significant load on the computer imple- 

FIELD OF THE INVENTION 5 me nting DLSW due to the need to maintain the connection 

This invention is directed to the field of computer com- state and protocol for both SNA and TCP sides of the 

munication networks. It is more particularly directed to connection. The DLSW scheme consists of encapsulating an 

inter-operation among different types of communication SNA packet into a TCP packet for carrying across the IP 

networks network. It is well-suited for sub-area SNA, but performs 

poorly when it is used for HPR networks. The congestion 

BACKGROUND OF THE INVENTION 10 contro l use d by RTP and TCP, the transport protocol used by 

The computer communication network in a typical enter- DLSW, do not operate well together, 

prise has changed substantially over the last two decades. An alternative way is to encapsulate SNA data over the 

Until the middle 80's most enterprise networks were based Internet UDP protocol. UDP is a unreliable connection-less 

on IBM's SNA architecture. Such networks connected data- 1S protocol and the encapsulation process can be done very 

base servers, usually IBM mainframes, to several client efficiently. However, SNA requires that the intermediate 

terminals. However, since the late 80's, the use of IP in the links provide reliable connectivity. This problem is 

enterprise network has become more prevalent. However, addressed by replacing the core SNA protocol in the main- 

the applications in the client and the mainframe are largely frame by the HPR (High Performance Routing) protocol, 

the original SNA-based ones. In such an environment, one 2Q which has been developed as an adjunct to APPN (Advanced 

needs to transport SNA application data over an IP network Peer to Peer Networking), an enhanced form of SNA pro- 

efficiently. The typical communication occurs between two tocol. 

SNA networks, which are connected by an intermediary IP HPR provides reliability by using a reliable transport 

network. protocol called RTP. RTP operates over unreliable links, and 

These computer communications networks in the enter- 2S uses a congestion control policy called ARB (adaptive 

prise interconnect these different terminals, work-stations, rate-based control). The ARB protocol sends out packets at 

personal computers, servers and host and enable them to a regular spacing, increasing the spacing when it detects that 

inter-operate. The inter-operation between these machines is the network is congested, and decreasing the spacing when 

obtained by using a common communication protocols and it senses that the network is lightly loaded, 

architecture. An architecture developed by IBM, called SNA 30 ARB was designed to operate on links which are used 

(Systems Networking Architecture) is a very common net- exclusively for HPR traffic. HPR assumes that each link on 

working architecture in an enterprise. the network is a constant-delay fixed-bandwidth point-to- 

Over the years, SNA architecture has evolved into several point link, on which all the traffic originates from competing 

versions. The initial version of SNA networks was relatively HPR sources. The characteristics of all links are determined 

static, and machine communication had to be pre- 35 when the link is brought up, and distributed to all other 

configured. This version is commonly known as the subarea nodes in the HPR network. When part of the network is 

SNA. The next version of an SNA network provided carried on an IP backbone, the entire IP network is made to 

dynamic configuration of communication among different appear like a single SNA link to the HPR/ARB algorithm, 

machines, and is known as APPN (Advanced Peer to Peer This is done by encapsulating HPR packets into UDP 

Networking). A further enhancement to APPN is known as 40 packets and tunnelling them through the IP network. 

HPR (High Performance Routing). HPR augments APPN by ARB fails to perform adequately in IP-based networks, 

providing two functional enhancements, a flexible routing ARB assumes that it is operating over a link with a fixed 

mode called ANR (Automatic Network Routing) and a delay and bandwidth. In reality, this virtual link consists of 

transport protocol called RTP. While subarea SNA required several links with competing traffic from other protocols 

that each of the links in the network be reliable, HPR does 45 such as TCP sharing the link bandwidth. The characteristics 

not have such a requirement. ANR links may be unreliable, of the link change over time. The performance of ARB is 

and RTP provides end-to-end reliability by retransmitting adversely affected when the link characteristics it assumes 

packets. differ substantially from the instant link characteristics. 

Another architecture, based on the Internet Protocol, or IP, While the creation of logical links is used in various 

is commonly used in an enterprise. IP provides a connec- 50 networking infrastructures, the performance of logical links 

tionless network on top of which two transport protocols are is generally poor. The key to good performance of HPR 

commonly supported, TCP, which provides a reliable byte- networks is the performance of its transport protocol RTP. 

stream transport of data end-to-end, and UDP which pro- RTP bases its operation on the behaviour of the intermediate 

vides an unreliable datagram service end-to-end. Due to links along the path of a pair of communicating hosts, and 

historical reasons, many eoterp rises have evolved to have a 55 assumes that all the links in the network are used exclusively 

structure in which clusters of SNA networks are inter- by RTP sources. In an HPR over IP environment, the logical 

connected with an intervening IP network. Communication links over the IP network carry a variety of non-RTP traffic 

in these networks usually tend to be from a client SNA which alters the characteristics of the logical link signifi- 

network which supports different terminals, to a SNA net- cantly. In this invention, we describe a solution which would 

work containing a mainframe server. The clieni networks are 60 cause RTP performance to improve when operating over an 

connected to the server networks by an intermediate IP IP backbone 

network. To the SNA network, the entire IP network appears A solution to the dynamic link configuration problem is to 

to be a set of SNA links that connect the different sites determine the right characteristics of the virtual link at 

together. SNA protocols operate transparently over these different times and distributing them to the network, 

virtual links. 65 However, such a dynamic distribution can cause a signiri- 

One way for transport of SNA traffic over IP network is cant overhead on the network. Furthermore, it will require 

using a protocol called DLSW (Data Link Switching). This changes to the HPR topology distribution protocol. 
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SUMMARY OF THE INVENTION 

The current invention describes a scheme by which the 
dynamic characteristics of the virtual links provided to the 
HPR network are fed back to the RTP protocol, and used to 
improve the performance of RTP traffic across the IP back- 
bone. The invention provides this feedback using the mecha- 
nisms provided by the HPR networking architecture, and 
thus can inter-operate with HPR networks that do not 
support this invention. 

In the present invention the link characteristics are deter- 
mined by programs running on both end of the IP network. 
These link characteristics are sent to the HPR end-node 
when an HPR connection is established. The HPR end-node 
obtains the information as appropriate fields in the Route- 
Setup Control Vector that is exchanged among different 
HPR nodes during the connection establishment phase. 

During the initiation of a connection, the HPR connection 
establishment protocols send out route-setup messages hop 
by hop along the desired path of the connection. When the 
message is received by the node along the path, it is required 
to send its fixed bandwidth and delay in the route -setup 
control vector, a component of the set-up message. This 
value is dynamically changed so as to reflect the current state 
of the network. 

There are many methods to obtain the current link char- 
acteristics of the multi-bop IP network. One method uses 
existing tools. There are existing tools that will identify the 
characteristics of an IP network. These tools impose a heavy 
load on the network for deciphering the path characteristics, 
but can be used at relatively long intervals to determine the 
path characteristics in a network. The commonly available 
such tools include and traceroute and pathchar. 

A second method uses a packet pair scheme. In this case, 
one end of the IP network sends out two back-to-back 
packets to its partner the other end of the network. The 
partner records the difference in time between the two 
packets, and relays the information back to the network. The 
information is relayed back to the originating node, which 
uses the temporal separation between the two nodes to 
determine the bottleneck link speed in the multi-hop net- 
work. The step is repeated several time so as to obtain a 
dynamic moving average. The network delay is obtained by 
averaging over the round-trip delays observed by the sender. 

A third method uses an aggressive congestion scheme. 
Here, one end of the IP network snoops into the RTP headers 
to determine whether a packet is being transmitted for the 
first time, is being re-transmitted, or an acknowledgment is 
being received for Ihe packet. The snooper computes the 
network bandwidth and delay parameters for ARB which 
would cause it to become as aggressive as the TCP conges- 
lion control scheme for the connection. This protocol 
requires a pre-existing RTP connection. In HPR, each node 
establishes a RTP connection between entities known as 
control points. The control point connection is an ideal 
connection to monitor for this comparison. 

Several of these schemes can be used together to obtain a 
good estimate of the current network characteristics. The 
estimation of dynamic link characteristics in this manner 
improves the performance of HPR ARB algorithm in a 
multiple-hop IP network, without requiring a change to 
existing APPN/HPR network protocols. 

One aspect of the present invention provides a method for 
dynamically determining link characteristics for a first com- 
munication protocol connection traversing a virtual commu- 
nication link. The virtual link forming a pathway over a 
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communication network supporting a second protocol. The 
method comprising: requesting a first set of link character- 
istics from a first node connected at a first boundary location 
along the network to a second node connected at a second 

5 boundary location along the network; sending a route setup 
message from at least one of the nodes along the pathway; 
receiving from each of the nodes a fixed bandwidth and 
delay in a response setup message for forming the first set of 
link characteristics; repeating the steps of requesting, send- 

10 ing and receiving at a plurality of predetermined times to 
obtain a new set of link characteristics at each of the 
predetermined times. In some embodiments the first proto- 
col is a HPR protocol, and the second protocol is an IP 
protocol. 

15 Another aspect of the present invention provides a com- 
munication apparatus comprising: a first communication 
engine having a first communication engine I/O, a second 
communication engine I/O, and a third communication 
engine I/O, the first communication engine I/O in order to 

20 accept a first protocol link and support a first protocol; an 
encapsulator having a first encapsulator I/O and a second 
encapsulator I/O, the first encapsulator I/O coupled to the 
second communication engine I/O; a network engine sup- 
porting a second protocol and having a first network engine 

25 I/O, a second network engine I/O, and a third network 
engine I/O; and an estimator having a first estimator I/O and 
a second estimator I/O, the first estimator I/O coupled to the 
second network engine I/O, and the second estimator I/O 
coupled to the second communication engine I/O. The first 

30 network engine I/O is coupled to the second encapsulator 
I/O, the third network engine I/O to accept a coupling to a 
network supporting the second protocol. The encapsulator 
converts data between the first and second protocols, and the 
estimator measures link characteristics of the communica- 

35 tion apparatus with respect to a remote apparatus coupled to 
the network. The network provides a remote virtual link with 
the apparatus. The virtual link is supported by the second 
protocol. In some embodiments the estimator measures the 
link characteristics at predetermined events. In some cases 

40 the predetermined intervals are fixed times increments rang- 
ing from a few milliseconds to many minutes. In specific 
embodiments, the first protocol engine is an HPR protocol 
engine and the second protocol engine is an IP protocol 
engine. 

45 Another aspect of the present invention provides a method 
for dynamically estimating the characteristics of a virtual 
HPR link. The method comprising of: sending a plurality of 
back-to-back packets from a first HPR/IP node to a second 
HPR/IP node; measuring packet separation data between the 

50 back-to-back packets when received at the second HPR/IP 
node; determining a set of link characteristics of the IP 
network from the packet separation data at the second node; 
and returning the link characteristics to the first HPR/IP 
node. In some embodiments the step of estimating is done 

55 repeatedly at a predetermined interval, and/or is comprised 
of sending a plurality of probe packets to a receiver at an end 
of the virtual link, and receiving a plurality of response 
packets from the receiver from which the set of link char- 
acteristics are determined. Sometimes the probe packets 

60 include the time of send data, and/or the response packets 
contain a delay characteristics of the virtual link, and/or ihe 
response packets contain bandwidth characteristics of the 
virtual link. 

Still another aspect of the present invention is to provide 
65 a method for dynamically estimating the characteristics of a 
virtual HPR link The method comprising: monitoring a 
plurality of requests and a plurality of acknowledgments 



12/02/2002, EAST Version: 1.03.0002 



US 6,215,772 Bl 

5 6 

flowing between a first HPR/IP node and a second HPR/IP second estimator cooperate in measuring the first set of 

node on a virtual IP link; correlating each acknowledgment characteristic links and the second set of characteristic links, 

with one request; determining a delay between the each and/or the first apparatus further comprises a first encapsu- 

acknowledgement and the one request; determining a con- lator to convert between the first and second protocols, and 

nection rate at which a first congestion control algorithm 5 the second apparatus further comprises a second encapsu- 

would transmit packets on a connection with the requests lator to convert between the first and third protocols, 

and the acknowledgments to the requests; and converting the i 0 stil i ano ther aspect, the present invention provides a 

connection rate into a corresponding link bandwidth and program storage device readable by machine, tangibly 

delay which would enable HPR to transmit at the same rate embodying a program of instructions executable by the 

as the first congestion control algorithm. In some specific 10 mac hine to perform method steps for dynamically estimat- 

embodiments the first congestion control algorithm is a TCP me characteristics of a virtual HPR link. The method 

congestion control algorithm, and the second congestion sleps comprising: monitoring a plurality of requests and a 

control algorithm is an ARB congestion control algorithm. plurality of acknowledgments flowing between a first HPR/ 

In some cases the estimator comprises: a monitor, which jp node and a se Con d HPR/IP node on a virtual IP link; 

observes the requests and acknowledgements/responses 15 correlating each acknowledgment with one request; deter- 

flowing through the apparatus; a correlator, which matches mining a delay between the each acknowledgement and the 

the requests to the acknowledgments/responses; a delay one request; determining a connection rate at which a first 

estimator, which determines the delay on a virtual HPR link congestion control algorithm would transmit packets on a 

on which the requests and acknowledgements/responses are connection with the requests and the acknowledgments to 

flowing; a TCP connection rate generator, which determines 20 the requests; converting the connection rate into a corre- 

a TCP rate at which a TCP connection sends data using a sp0 nding link bandwidth and delay which would enable 

TCP congestion control algorithm; an ARB connection rate H p R t0 transmit at the same rate as the first congestion 

generator, which determines an ARB rate at which the TCP control algorithm. 

connection would send data using an ARB congestion fa ^ mo{het ' iy the present invention provides an 
control algontto; a lin^ 25 aflicle of manufacture comprising: a computer usable 
equal to the TCP rate if the TCP rate is higher than the ARB medium ^ readab i e program code embodied 
rate, otherwise sets the link rate to the ARB rate. In some ^ cau&e ^ d fc ^ timatation of link character- 
embodiments the estimator comprises: a monitor, which ^ of & HpR lmk Thc readablc program 
observes the requests and acknowledgements/responses ^ m ^ ^ rf mamifacturc computer 
flowing through the apparatus; a correlator, which matches 30 readable Q code tQ cause a computer to effect: the 
the requests to the acknowledgments/responses; a t delay monitori of a lura]it of te and a plurality of 
estimator, which determines the delay on a virtual HPR link acknowledgme nts flowing between a first HPR/IP node and 
on which the requests and acknowledgemente/responses are a HpR/Ip node on a virtua] IP ^ the correlating of 
flowing; a first connection rate generator, which determines eacfa acknowledgment ^ one request; the determining of 
a first rate at which a connection sends data using a first 35 a del betweea ^ each acknowledgement and the one 
congestion control algorithm; a second connection rate reque5t; me detennmmg of a connection rate at which a first 
generator, which determines a second rate at which the congestion control algorithm would transmit packets on a 
connection would send data using a second congestion COQnection ^ me sts and the acknow ledgments to 
control algorithm; a link rate generator, which sets a link rate ^ ^ ^ ^ of thc rate mto 
equal to me first rate if the first rate is higher than the second 40 a link baQ dwidth and delay which would 
rate, otherwise sets the link rate to the second rate. eQable HpR tQ Umsmii at the ^ ra!e as the first 

In still another aspect of the present invention, a gateway tion algorithm, 
system is provided which comprises: a first apparatus having 

a first apparatus primary I/O and a first apparatus secondary DESCRIPTION OF FIGURES 

I/O, and a second apparatus having a second apparatus 45 

primary I/O and a second apparatus secondary I/O. The first These and other objects, features, and advantages of the 
apparatus primary I/O to accept a first link connection present invention will become apparent upon further con- 
supporting a first protocol, and the first apparatus secondary sideration of the following detailed description of the inven- 
I/O to accept a connection to a communications network tion whea read m conjuncUon with the drawing figures, in 
supporting a second protocol. The second apparatus primary 50 which: 

I/O to accept a second link connection supporting a third FIG. 1 shows several clusters of HPR networks connected 

protocol, and the second apparatus secondary I/O to accept together by an IP backbone network which is accessed via 

a connection to the communications network. The second HPR-IP gateways; 

apparatus is coupled to the first apparatus via a virtual link pjo, 2(a) shows the structure of any IP packet on an IP 

within the communications network, with the virtual link 55 network; 

supporting the second protocol. The first apparatus com- FIG. 2(b) shows the structure of a packet on a pure HPR 

prises a first estimator to measure a set of first link charac- network; 

teristics of the virtual link with respect to the second p]G ^ s ^ ^ of a HpR ^ whefi ft fe 
apparatus, and wherein. The second apparatus comprises a carried ^ ( he IP network- 
second estimator to measure a set of second link character- 60 ' Imn In ... 
istics of the virtual link with respect to the first apparatus. In ™. 3 shows the structure of an HPR-IP gateway illus- 
some case, the first estimator measures the first link char- the components that make up the HPR-IP gateway; 
acteristics at predetermined intervals. In some cases the FIG. 4 shows a flow-chart used by the KTP source engine 
second estimator measures the second link characteristics at to determine the characteristics of a link along its path 
the predetermined intervals, and/or the first and third pro- 65 during its congestion control scheme; 
tocols are HPR protocols and the second protocol is an IP FIG. 5 shows a flow-chart used by the esiimator compo- 
protocol. In some embodiments the first esiimator and the nent of the HPR/IP gateway to estimate the dynamic char- 
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acteristics of a virtual link along its path using pre-existing 
link characteristic tools; 

FIG. 6 show the flow-chart used by the estimator com- 
ponent of the HPR/IP gateway to request the estimation of 
the dynamic characteristics of a virtual link along its path by 5 
sending a group of probe packets in the network; 

FIG. 7 show the flow-chart used by the estimator com- 
ponent of the HPR/IP gateway to respond to the estimation 
of the dynamic characteristics of a virtual link along its path 
using the method of sending a group of probe packets in the 10 
network; 

FIG. 8 shows the flow-chart used by the estimator com- 
ponent of the HPR/IP gateway to estimate the dynamic 
characteristics of a virtual link along its path using a third ^ 
method; and 

FIG. 9 shows the structure of an estimator component of 
the HPR/IP gateway which estimates the dynamic charac- 
teristics of a virtual link along its path using a method of 
snooping into the higher level protocol headers at the 2Q 
HPR-1P gateway. 

DETAILED DESCRIPTION OF THE 
INVENTION 

An environment in which the invention can be used is 25 
shown in FIG. 1. It shows three HPR networks, 100, 110 and 
120, which are interconnected together by a backbone IP 
network 130. The IP backbone 130 is connected to the HPR 
network 100 by means of an HPR-IP gateway 140. 
Similarly, HPR network 110 is connected by means of an 3(J 
HPR-IP gateway 150, and HPR network 120 is connected to 
the backbone by means of an HPR-IP gateway 160. 

The task of the HPR-IP gateways is to provide the 
mechanisms so that the HPR networks 100, 110 and 120 can 
inter-operate. The gateways perform this task by creating 35 
logical links across the IP backbone network that provide the 
interconnection between the different HPR networks. These 
logical links are created by establishing a communication 
channel across the IP network using the UDP protocol. The 
creation of logical links is used widely within the network- 40 
ing community. FIG. 1 shows three logical links, 170, 180 
and 190 which connect the three HPR networks together. In 
general, a virtual link may include several IP nodes and/or 
links. 

FIGS. 2(a), 2(6) and 2(c) show the structure of packets as 45 
they are carried over different parts of the network. A packet 
on the IP backbone consists of two parts, an IP header 210 
and an IP data 220, as shown in FIG. 2(a). A packet on an 
HPR network, e.g. in network 110 of FIG. 1, has the general 
format shown in FIG. 2(6). It consists of an ANR header 230 50 
followed by an RTP header 240. These two headers are 
followed by the payload of RTP, or RTP data 250. When the 
packet shown in FIG. 2(6) is encapsulated over the virtual 
link, it is transported in the format shown in FIG. 2(c), which 
consists of an IP header 260 followed by the ANR header 55 
270 and the RTP header 280, and the RTP data 290. At the 
point of origin of a virtual link, the ANR header 270 is 
derived from the ANR header 230, the RTP header 280 is 
derived from the ANR header 250, and the RTP data 290 is 
derived from the RTP data 250. At the destination point of 60 
the virtual link, the IP header 260 is removed, and the 
original packet in the format of FIG. 2(6) is regenerated, 
with the ANR header 230 derived from ANR header 270, 
RTP header 240 derived from RTP header 280, and RTP data 
250 derived from RTP data 290. 65 

The performance of a logical link created for intercon- 
necting different HPR networks is very variable and unpre- 
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dictable. This causes performance problems for RTP, which 
is the transport protocol used with HPR networks. RTP is 
used for point-to-point communication between a pair of 
applications on two computers connected together by an 
HPR network. RTP bases its reaction to congestion control 
on the notion of the bottleneck link, that it the slowest link 
between a pair of communicating links. It also assumes that 
the links are going to be used exclusively by RTP sources. 

Because of competition from competing traffic from the 
network, the characteristics of the virtual link over the IP 
backbone varies dynamically over time. Sometimes, it may 
be the slowest link along the path, while at other times it may 
not be. Furthermore, when it is the slowest link on the path, 
the bandwidth available for HPR traffic varies dynamically 
due to the presence of competing non-HPR traffic. 

In order for RTP to operate efficiently in this environment, 
the source needs to obtain the information about the 
dynamic characteristic of the link and use that in its con- 
gestion control algorithm. However, this information needs 
to be provided in a manner that is transparent to RTP so that 
the same RTP engine can be used for HPR links that are 
encapsulated over IP as well as native HPR links. In this 
invention we propose that this adaptation be provided by the 
HPR-IP gateway. 

FIG. 3 shows a structure of an HPR-IP gateway that uses 
information about the current characteristics of the virtual 
link to improve the performance of HPR encapsulation over 
IP. The HPR-IP gateway 300 consists of four components. 
The HPR engine 320 is responsible for receiving and 
transmitting HPR packets in the format shown in FIG. 2(6), 
and for interacting with other components of the HPR 
network that send packets to it. The IP engine 340 is 
responsible for receiving and transmitting packets in the IP 
backbone network in the format shown in FIG. 2(d). The 
encapsulator 330 is responsible for converting packets from 
the HPR format to the IP format and vice-versa. The 
estimator 310 is a module that monitors the performance of 
the IP network, and computes its current link characteristics. 
The estimator determines the current link characteristics by 
monitoring the IP packets received and transmitted by the IP 
engine 340, and the RTP engine 320. It also communicates 
with the RTP engine to indicate the current characteristics of 
the virtual link. The encapsulator 330 can add information 
during the encapsulation and decapsulation phase, which 
would enable the estimator 310 to obtain better estimates of 
the link performance. One example of such information 
would be the time at which a packet is received for encap- 
sulation. The estimator 310 at the other end looks at this 
information in order to determine the performance of the 
network. The encapsulators in the network can also 
exchange special probe packets in order to determine the 
network performance. These special probe packets carry 
time-stamps and sequence number information which per- 
mits the encapsulator at the other end to determine network 
performance. 

The information about the current link characteristics 
need to be conveyed back to the RTP engine at the com- 
municating node. As per the HPR architecture, the source 
RTP initiates a connection by sending out a route- 
establishment message which determines the characteristics 
of the path between the pair of communicating nodes. The 
current characteristics of the virtual path are always pro- 
vided by the HPR engine in the HPR-IP engine gateway. 
Thus, each RTP connection that is initiated obtains the 
characteristics of the virtual links at the start of the connec- 
tions. 

For RTP connections that are relatively small, the 
dynamic determination of link characteristics at the time of 
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connection establishment is adequate for the duration of the existing tool are converted to the link characteristics under- 
connection. For long-lived connections, the link character- stood by the HPR algorithm. Specifically, the link delay and 
istics vary over the life of the connection. The RTP engine bandwidth metrics are obtained from the output of the 
at the source node can account for it by sending route- existing tools. After that, in step 507, these link character- 
establishment queries at periodic intervals. However, this 5 istics are stored in a location that is accessible by the RTP 
would require a change to the existing HPR architecture. source engine executing the flow-chart described in FIG. 4 
Alternatively, the same RTP end-point is likely to have and the procedure is exited 509. 

several RTP connection establishments over a period of Method 2 consists of sending a group of probe packets on 

time. The virtual link would be shared among all the the network. These probe packets are sent out by an esti- 

different RTP connections established by the node. The RTP 1Q mator in an originating HPR-IP gateway to the estimator in 

end-point would use the most current link characteristics in another HPR-IP gateway. The sending estimator puts the 

order to drive its congestion control scheme. The most current time-stamp and sequence number in the origin 

current link characteristics would be obtained by the most packet. This information is analyzed by the recipient esti- 

current response to a route establishment message that mator who can analyze this information locally, or can store 

involves the virtual link. Links characteristics obtained as a 15 its current time-stamp into the packet and reflect it back to 

result of different route establishment messages can be the original HPR/IP gateway. The analysis of the informa- 

correlated easily because they have a common ANR path tion determines the delay and bandwidth of the virtual link 

from the source. The most current link characteristics can by, for example, observing the sequence numbers and the 

also be broadcast on the HPR network by the estimator in time-stamp placed by the originating HPR-IP gateway. 

HPR-IP gateway on a periodic basis, or when the charac- 2Q FIGS. 6 and 7 show the flow-charts used by an estimator 

teristics change by a significant amount. The broadcasts component of the HPR/IP gateway to obtain the dynamic 

contain an identifier for the link whose characteristics are characteristics of a virtual link along its path using Method 

contained in the broadcast. 2. Method 2, requires two estimators, one on each of the two 

FIG. 4 shows a flow-chart used by an RTP source engine ends of a virtual link enabling the HPR/IP gateways to 

to determine the characteristics of a link along its path 25 communicate with each other. One of these estimators 

during its congestion control scheme. The procedure is initiates the estimating procedure using the flow-chart illus- 

entered in step 400 whenever the characteristics of a link trated in FIG. 6. The other estimator responds to the esti- 

have to be determined. In step 410, the algorithm verifies mation requests using the flow-chart illustrated in FIG. 7. 

whether a new link characteristic is available for the link in The procedure illustrated in FIG. 6 is generally entered at 

question. The new link characteristic may be available as a 30 pre-determined times 601. In step 603, two or more packets 

result of either a new route establishment message using the are prepared for transmission to the estimator on a second 

same link, or as a result of a new link status update broadcast HPR/IP gateway. In this preparation step, the current time 

by the HPR-IP gateway. If a new link characteristic is and packet sequence number is put into the packet. After the 

available, it is stored locally 420. In step 430 of the packets have been prepared, they are shipped out on the IP 

algorithm, the stored link characteristics are extracted. The 35 gateway sequentially 605. Subsequently, in step 607, the 

stored data are always the most recently updated data. procedure waits for the responses/acknowledgments to the 

Finally, the algorithm exits in step 450. packets to be received, and stores them in local memory. In 

In accordance with the present invention, the link char- step 609, the procedure checks if it has obtained the required 

acteristics of the virtual link may themselves be identified by number of responses to determine the link characteristics. If 

using a variety of techniques. Three such techniques that can 49 the required responses have been received, it computes the 

be used for determining the link characteristics of an IP link characteristics on the basis of those responses in step 

virtual link are described herein. Other techniques may be 613. In step 615, these responses are stored so that they can 

similarly employed to implement the present invention. be used by an RTP source engine procedure such as that 

Method 1, consists of using pre-existing link character- illustrated in FIG. 4. The process then stops in step 619. If 

istic tools. These tools (e.g. a tool known as pathchar on the 45 the result of step 609 is negative, and all required responses 

Internet) send out special IP control messages that become have not been received, the procedure checks if a specified 

invalid after a fixed number of hops. On the expiration of the time-out period after the requests have been sent has elapsed 

invalid messages, the corresponding router sends a reply 611. If the time-out period has elapsed, and the response 

back to the originating host. This scheme identifies the have not been received, the estimation procedure stops 617. 

characteristics of all the IP links along the path. The indi- 50 However, if the time-out period has not elapsed, the proce- 

vidual path characteristics are aggregated together by adding dure continues to wait for more responses, 

up their delays and taking the minimum available bandwidth The procedure illustrated in FIG: 7 is entered by an 

on the links. This information is then provided to the HPR estimator during its initialization in step 701. In step 703, the 

engine. Other tools available for this purpose are bprobe and estimator waits for probes from its partner estimator on the 

cprobe, developed at Boston University. These tools are 55 other end of the virtual link. When the probe packet is 

invoked at periodic intervals, as well as when an abrupt received, the packets are processed in step 705. Depending 

change in network conditions is observed. The abrupt on the probing method selected by the two estimators, the 

change in network conditions include a sudden loss of processing in step 705 can either simply reflect the probe 

packets in the network, or a failure of a link or a node in the packets back to the sender, or it can compute the link 

network. 60 characteristics at its local end. On each probe packet, the 

FIG. 5. shows a flow-chart used by an estimator compo- procedure checks if a response packet needs to be generated, 

nent of a HPR/IP gateway to obtain the dynamic character- This check is performed in step 707. If no response is 

istics of a virtual link along its path using Method 1. The needed, the procedure goes back to receiving more packets 

procedure is initiated at predetermined times. The procedure in step 703. If the response is needed, a response packet is 

is entered in step 501. In step 503, an existing tool, such as 65 generated and sent back in step 709, after which the esti- 

traceroute or bprobe, is invoked to obtain the characteristics mator procedure reverts back to receiving the next probe 

of the virtual link. In step 505, the results obtained from the packet in step 703. 
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Method 3 consists of snooping into the higher level 
protocol headers at the HPR-IP gateway. The HPR-IP gate- 
way traditionally looks only at the IP and ANR headers of 
packets shown in FIG. 2. RTP header contains various 
congestion control information data that can be used to 
determine the rate at which packets should be sent in the 
network. The RTP protocol uses a particular algorithm to 
translate this congestion control information into a trans- 
mission rate for a connection. Other congestion control 
protocols, such as TCP use somewhat different algorithms to 
translate the congestion control information into the trans- 
mission rate. The HPR-IP gateway analyzes the congestion 
control information and uses both the TCP and RTP con- 
gestion control algorithms to determine the transmission 
rate. If the transmission rate for TCP turns out to be higher, 
the HPR-IP gateway computes the effective bandwidth and 
delay of the virtual link for which the RTP congestion 
control scheme would send packets at the same transmission 
rate as the TCP algorithm. These link characteristics are then 
published by the HPR-IP gateway as those of the virtual link. 

FIG. 8 shows the flow-chart used by the estimator com- 
ponent of the HPR/IP gateway to obtain the dynamic char- 
acteristics of a virtual link along its path using Method 3. 
The procedure is entered at step 801 when an HPR/IP 
connection is established through the HPR/IP gateway. At 
this step, a default set of link characteristics are assumed for 
the virtual link. In step 803, the packets flowing through the 
HPR/IP gateway are monitored and classified into packet 
transmissions, acknowledgments of packet transmission, 
requests for network rate information, and replies 
(acknowledgment) of network rate information. In step 804, 
the procedure checks if the packet signals the end of the 
HPR/IP connection. If so, the procedure exits in step 821. 
Otherwise, In step 805, the procedure checks if the packet is 
an acknowledgment (or rate reply packet). Information from 
the rate requests and packet transmissions which is needed 
for computing link characteristics is stored for further pro- 
cessing in step 807. When an acknowledgment packet, or 
reply to a rate request is obtained, in step 809, the procedure 
finds its corresponding rate request or original packet. In 
step 811, the time when the packet was originally transmit- 
ted and when the reply (or acknowledgment) are received 
are compared, and the delay on the virtual HPR link is 
estimated from the difference in the two times. In step 813, 
the information about the packets and the acknowledgment 
patterns they received are analyzed to determine the rate that 
a TCP congestion control algorithm would obtain when used 
for the specified HPR/IP connection. This rate is obtained by 
executing the procedures of the TCP congestion control 
algorithm, which is well known in prior art. In step 815, the 
ARB congestion control algorithm is executed and the rate 
it would assign to the connection under the same set of 
constraints is computed. The ARB protocol engine is well 
known in prior art. One distinguishing feature about the TCP 
congestion control algorithm is that it computes its connec- 
tion rate without using the characteristics of the virtual link, 
whereas ARB protocol uses the link characteristics as an 
input in computing its connection rate. In step 817, the TCP 
connection rate and the ARB connection rate are compared. 
If the ARB connection rate is higher than the TCP connec- 
tion rate, the link characteristics are left unchanged, and the 
procedure reverts to step 803. If the TCP connection rate is 
higher than the ARB connection rate, then in step 819 the 
procedure computes the virtual link rate which would cause 
ARB to operate at the same connection rate as TCP. One 
method of doing it is to divide the TCP connection rate by 
an configuration parameter of ARB which specifies the ratio 
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of ARB initial connection rate to the link rate. This infor- 
mation is stored and the procedure moves back to step 803. 

Generally, a request is information that is generated by 
one of end of a connection and sent to the other end. An 

5 acknowledgment is an indication from the other end that it 
received the request. After the second end has processed the 
request, it may also send a reply back to the first end. The 
TCP congestion control algorithm determines a rate at which 
to send new requests by analyzing the acknowledgments 

10 received for requests sent earlier. The ARB congestion 
control algorithm determines a rate at which to send new 
requests by analyzing the replies generated to a subset of 
requests sent earlier. Both of these, the TCP acknowledg- 
ments and the ARB replies, are herein commonly referred to 

15 as acknowledgments/responses. 

FIG. 9 shows a flow structure of the estimator that can be 
used to determine the dynamic characteristics of a virtual 
HPR link. The estimator 901 consists of packet monitor 903, 
a correlator 905, a delay computer 907, a first connection 

20 rate computer 909, a second connection rate computer 911 
and a link rate computer 913. The packet monitor 903 
receives information about packets flowing through the 
HPR/IP gateway on a monitor input port, and classifies the 
packets into requests, acknowledgments, replies and packet 

25 transmissions. The packet monitor 903 passes the classifi- 
cation information to a correlator 905 via a correlator output 
port. The correlator 905 receives this information, and sends 
information about acknowledgments and corresponding 
packets (or matching request-reply pairs) to the delay com- 

30 puter 907 via the delay output interface. The delay computer 
907 receives this information via its correlator input inter- 
face and determines the delay of a virtual HPR link from this 
information. The delay computer 907 outputs the link delay 
on a link delay output port The correlator 905 sends the 

35 packet information to a first rate computer 909 and a second 
rate computer 911. The first connection rate computer deter- 
mines the connection rate for a HPR/IP session using the 
TCP congestion control algorithm, while the second con- 
nection rate computer determines the connection rate for a 

40 HPR/IP session using the ARB congestion control algo- 
rithm. The two connection rates thus computed are sent to a 
link rate computer 913. The link rate computer compares the 
two connection rates, and chooses the higher of the two 
rates. If the TCP rate is not the highest, then the link rate 

45 computer determines the minimum link rate at which the 
second connection rate computer 911 would have obtained 
the same rate as the first connection rale computer 909. The 
computed link rate is sent on a link rate output port. 
All the embodiments of the present inventions are also 

50 suitable for inclusion in an article of manufacture and/or a 
program storage device readable by machine, with the 
article/device being used for implementing the present 
invention in a particular system. For example, the article 
may be a floppy or hard disc, a CDROM or any media and/or 

55 equipment capable of being loaded with data or a program 
for enabling a system or computer to perform an embodi- 
ment of the invention. For example this may be any of the 
HPR gateways 100 shown in FIG. 1. This includes embodi- 
ments wherein the data and/or programs are available loca- 

60 tions that are remote from the network and/or elements that 
employ this invention. This encompasses cases in which the 
data and/or programs are downloaded from one location/ 
medium for use at another location or another medium. 
It is noted that these schemes are only example of the 

65 different methods and apparatus that can be used. One 
versed in the art can replace them with other schemes for 
predicting the characteristics of the virtual link. It is further 
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noted that although the description is made for particular 
arrangements and steps, the intent and concept of the present 
invention are suitable and applicable to other arrangements 
and combination of steps. It will be clear to those skilled in 
the art that other modifications to the disclosed embodiments 
can be effected without departing from the spirit and scope 
of the invention. 
What is claimed is: 

1. A method for dynamically estimating the characteristics 
of a virtual High Performance Routing (HPR) link, said 
method comprising: 
sending a plurality of back-to-back packets from a first 
High Performance Routing over the Internet Protocol 
(HPR/IP) node to a second HPR/IP node; 
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a network engine supporting a second protocol and having 
a first network engine I/O, a second network engine 
I/O, and a third engine I/O, a second network engine 
I/O, and a third network engine I/O, said first network 
engine I/O coupled to said second encapsulator I/O, 
said third network engine I/O to accept a coupling to a 
network supporting said second protocol; and 
an estimator having a first estimator I/O and a second 
estimator I/O, said first estimator I/O coupled to said 
second network engine I/O, and said second estimator 
I/O coupled to said second communication engine I/O, 
wherein said encapsulator converts data between said first 
and second protocols, and said estimator measures link 
characteristics of said communication apparatus with 



measuring packet separation data between the back-to- j5 respect to a remote apparatus coupled to said network, said 



back packets when received at the second HPR/IP 
node; 

determining a set of link characteristics of the Internet 
Protocol (IP) network from the packet separation data 
at the second node; and 

returning the link characteristics to the first HPR/IP node. 

2. A method as in claim 1, wherein the step of estimating 
is done repeatedly at a predetermined interval. 

3. A method as in claim 1, wherein the step of estimating 
is comprised of sending a plurality of probe packets to a 
receiver at an end of said virtual link, and receiving a 
plurality of response packets from the receiver from which 
the set of link characteristics are determined. 

4. A method as in claim 3, wherein the probe packets 
include the time of send data. 

5. A method as in claim 3, wherein the response packets 
contain a delay characteristics of the virtual link. 

6. A method as in claim 3, wherein the response packets 
contain bandwidth characteristics of the virtual link. 

7. A method for dynamically estimating the characteristics 
of a virtual High Performance Routing (HPR) link, said 
method comprising: 

monitoring a plurality of requests and a plurality of 
acknowledgments flowing between a first High Perfor- 
mance Routing over the Internet Protocol (HPR/IP) 
node and a second HPR/IP node on a virtual Internet 
Protocol (IP) link; 

correlating each acknowledgment with one request; 

determining a delay between said each acknowledgment 
and said one request; 

determining a connection rate at which a first congestion 
control algorithm would transmit packets on a connec- 
tion with the requests and the acknowledgments to the 
requests; and 

converting the connection rate into a corresponding link 59 
bandwidth and delay which would enable HPR to 
transmit at the same rate as the first congestion control 
algorithm. 

8. A method as described in claim 7, wherein the first 
congestion control algorithm is a Transmission Control 
Protocol (TCP) congestion control algorithm, and the sec- 
ond congestion control algorithm is an Adaptive Rate-Based 
(ARB) congestion control algorithm. 

9. A communication apparatus comprising: 
a first communication engine having a first communica- 
tion engine I/O, a second communication engine I/O, 
and a third communication engine I/O, said first com- 
munication engine I/O lo accept a first protocol link and 
support a first protocol; 

an encapsulator having a firsl encapsulator I/O and a 
second encapsulator I/O, said first encapsulator I/O 
coupled to said second communication engine I/O; 



network providing a remote virtual link with said gateway 
apparatus, said virtual link supported by said second 
protocol, and 

wherein the estimator comprises: 

a monitor, which observes the requests and 
acknowledgments/responses flowing through the appa- 
ratus; 

a correlator, which matches the requests to the 

acknowledgments/responses; 
a delay estimator, which determines the delay on a virtual 
HPR link on which the requests and acknowledgments/ 
responses are flowing; 
a TCP connection rate generator, which determines a TCP 
rate at which a TCP connection sends data using a TCP 
congestion control algorithm; 
an ARB Connection Rate Generator, which determines an 
ARB rate at which the TCP connection would send data 
using an ARB congestion control algorithm; 
a link rate generator, which sets a link rate equal to the 
TCP rate if the TCP rate is higher than the ARB rate, 
otherwise sets the link rale to the ARB rate. 
10. A communication apparatus comprising: 
a first communication engine having a first communica- 
tion engine I/O, a second communication engine I/O, 
and a third communication engine I/O, said first com- 
munication engine I/O to accept a first protocol link and 
support a first protocol; 
an encapsulator having a first encapsulator I/O and a 
second encapsulator I/O, said first encapsulator I/O 
coupled to said second communication engine I/O; 
a network engine supporting a second protocol and having 
a first network engine I/O, a second network engine 
I/O, and a third network engine I/O, said-first network 
engine I/O coupled to said second encapsulator I/O, 
said third network engine I/O to accept a coupling to a 
network supporting said second protocol; and 
an estimator having a first estimator I/O and a second 
estimator I/O, said first estimator I/O coupled to said 
second network engine I/O, and said second estimator 
I/O coupled to said second communication engine I/O, 
wherein said encapsulator converts data between said first 
and second protocols, and said estimator measures link 
characteristics of said communication apparatus with 
60 respect to a remote apparatus coupled to said network, said 
network providing a remote virtual link with said gateway 
apparatus, said virtual link supported by said second 
protocol, and 

wherein the estimator comprises: 
65 a monitor, which observes the requests and 
acknowledgments/responses flowing through the appa- 
ratus; 
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a correlator, which matches the requests to the 
acknowledgments/responses; 

a delay estimator, which determines the delay on a virtual 
HPR link on which the requests and acknowledgments/ 
responses are flowing; 

a first connection rate generator, which determines a first 
rate at which a connection sends data using a first 
congestion control algorithm; 

a second connection rate generator, which determines a 
second rate at which the connection would send data 
using a second congestion control algorithm; 

a link rate generator, which sets a link rate equal to the 
first rate if the first rate is higher than the second rate, 
otherwise sets the link rate to the second rate. 

11. A program storage device readable by machine, tan- 
gibly embodying a program of instructions executable by the 
machine to perform method steps for dynamically estimat- 
ing the characteristics of a virtual High Performance Rout- 
ing (HPR) link, said method steps comprising: 

monitoring a plurality of requests and a plurality of 
acknowledgments flowing between a first High Perfor- 
mance Routing over the Internet Protocol (HPR/IP) 
node and a second HPR/IP node on a virtual Internet 
Protocol (IP) link; 

correlating each acknowledgment with one request; 

determining a delay between said each acknowledgment 
and said one request; 

determining a connection rate at which a first congestion 
control algorithm would transmit packets on a connec- 
tion with the requests and the acknowledgments to the 
requests; 
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converting the connection rate into a corresponding link 
bandwidth and delay which would enable HPR to 
transmit at the same rate as the first congestion control 
algorithm. 

5 12. An article of manufacture comprising: 

a computer usable medium having computer readable pro- 
gram code embodied therein to cause the dynamic estima- 
tion of link characteristics of a virtual High Performance 
10 Routing (HPR) link, the computer readable program code in 
said article of manufacture comprising computer readable 
program code to cause a computer to effect: 

the monitoring of a plurality of requests and a plurality of 
acknowledgments flowing between a first High Perfor- 
mance Routing over the Internet Protocol (HPR/IP) 
node and a second HPR/IP node on a virtual Internet 
Protocol (IP) link; 
the correlating of each acknowledgment with one request; 
the determining of a delay between said each acknowl- 
edgment and said one request; 
the determining of a connection rate at which a first 
congestion control algorithm would transmit packets 
on a connection with the requests and the acknowledg- 
ments to the requests; and 
the converting of the connection rate into a corresponding 
link bandwidth and delay which would enable HPR to 
transmit at the same rate as the first congestion control 
algorithm. 
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